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purpose, and use of evaluation results, was lacking. As a result, many 
participants thought that at present there were neither rewards nor 
consequences related to evaluation other than their own satisfaction in 
learning how they might improve programs or their success in gaining 
additional grant funds for program support. The perceptions and attitudes of 
the faculty based on campus were generally less favorable than those of 
Extension educators who participating by teleconference. (SLD) 
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University of Minnesota Extension Faculty and Staff Members’ Attitudes 
Toward and Perceptions About Program Evaluation: An Organizational 
Concern in Relation to Accountability Demands 

Marilyn Grantham, Accountability and Reporting Specialist 

Some writers and thinkers in the evaluation profession are calling this era 
“the age of accountability” because of the growing demand from public and 
nonprofit funding sources for a strict accounting of how funds are spent and what 
is accomplished with them. Because of their unique collaborative funding base, 
cooperative Extension organizations, in particular, must respond to a number of 
funders — federal, state, and county governments, as well as grantors and 
contractors. This situation raises several issues. For one, the demand for 
accountability is coming at a time when organizational resources in general are 
stable at best and no additional funds are being provided specifically for 
accountability. So responding to accountability demands and preparing and 
reporting data, including the outcomes and impacts of educational programs, 
takes organizational resources that otherwise would be invested elsewhere, such 
as in program development and evaluation. Secondly, accountability enhances 
the need for program evaluation in order to provide reliable and valid information 
about outcomes and impacts. The end result is a growing need for more 
program evaluations at a time when staff time and additional money are 
becoming scarcer. 

Some background on this study : 

In late 1996, Robert Shumer, a faculty member in the College of Education and 
Human Development, and Richard Krueger, Program Evaluation Leader, 
approached the University of Minnesota Extension Service’s dean and director 
with a proposal to conduct a series of "hands on” non-credit staff development 
workshops on outcome evaluation. Some funds were designated and the author 
of this paper volunteered to assist Professors Shumer and Krueger with the 
workshop series. 

The target audience was faculty arid academic staff who had received 
competitive Extension internal grants during 1995 and 1996 — the first two years 
that this approach for providing “start-up” funding for new and innovative 
programs had been used. This audience was chosen because one of the 
specifications associated with receiving an internal grant was that the recipient(s) 
would evaluate their work and report the results to the members of the review 
committee. Extension administrators, and their colleagues. 

Members of the target audience were sent personalized letters of inyitation from 
the dean and director, inviting them to participate in the outcome evaluation 



workshop series. Grantees were told in the invitation letter that the purpose of 
the workshop series was to help them design and conduct an evaluation of their 
program. 

Each workshop in the series of four was planned "for five hours per day on 
campus and spaced about two months apart so that participants would have time 
to put into effect what they’d learned before coming back for the next phase. 
Training materials and lunches were provided and travel expenses for staff off 
campus were covered from the grant funds. 

About 30-32 internal grant recipients indicated an interest in the workshop series. 
About 24-25 of them came to the initial workshop. When the second workshop 
was held about two months later, the number of participants dwindled to about 
two-thirds of the initial group. The number of participants dropped substantially 
again for the third workshop. About a third of the initial number of participants 
were in three or four of the workshops. However, those participants who 
persisted indicated that they were highly pleased with the learning process and 
the outcomes in terms of designing and implementing evaluation plans in an oral 
evaluation at the end of the final workshop. 

Nevertheless, this experience raised a number of issues in the author’s mind 
about Extension faculty and staff attitudes toward and perceptions of program 
evaluation, as well as howto successfully design and deliver staff development 
that would provide the necessary skills and competencies to do program 
evaluations. It was this situation that “set the stage" for conducting a series of 
five focus groups with Extension faculty and staff in order to explore their 
attitudes and perceptions about program evaluation. 

Research methodology : 

The names of faculty and staff members invited to participate in the focus groups 
were taken from the original list of faculty and staff who had received Extension 
internal grants during 1995 and 1996. The invitees were those persons on the 
list who had not responded to the invitation to participate in the outcome 
evaluation workshop series. Subtracting faculty and staff members who had 
come to one or more of the workshops resulted in 55 persons who were 
contacted about participating in a focus group. A total of 35 faculty and staff 
members participated in one of five focus groups. Twenty-two of them were 
Extension educators who participated in one of three groups held via 
teleconferences. The remaining 13 persons participated in one of two face-to- 
face focus groups held on campus. Six of these persons are faculty members in 
academic departments; the remaining seven persons are academic professional 
staff on campus. 

The purpose of the focus groups was to explore Extension faculty and staff 
attitudes toward program evaluation and their perceptions of their individual 




2 



4 



responsibility for evaluating programs, the benefits and the barriers in evaluating 
programs, and the support and recognition provided by Extension for this kind of 
activity. The questions asked in all five focus groups evolved from themes that 
emerged out of the evaluative discussion held in the final Outcome Evaluation 
Workshop in October 1997. This discussion had been videotaped and the tape 
observed for themes, then an edited version was produced as the evaluation 
report for the workshop series. 

The questions asked participants in all five focus groups were 

1 . When you hear the words “program evaluation,” what immediately comes to 
your mind? 

2. When you are asked to evaluate a program or a project, what do you think 
that means? What are you expected to do? 

3. What are the benefits to evaluating Extension programs? 

4. What are the barriers to evaluating Extension programs? 

5. What have you done about evaluating the program for which you received an 
Extension internal grant? 

6. What would have helped you with this effort or any other program evaluation? 

7. How should Extension recognize and reward evaluation work? 

Issues emerging from the answers to these questions : 

Evaluation issues: 

• Difficulty measuring outcomes and impacts, particularly in the short-term 
when Extension programs often won’t show such results for many years. 

• Difficulty in separating Extension’s role in achieving outcomes from other 
influences on program participants’ attitudes or behaviors. 

• What is “success” in terms of program outcomes? 

• Assessing “unintended consequences” of programs. 

• Lack of knowledge/connection with evaluation studies done elsewhere — 
models that might be adapted for use in Minnesota. 

• Tendency to focus on numbers and other “input” information that is more 
easily gathered, rather than on the “so what” question. 

• Need for new approaches to program evaluation that are “culturally” 
appropriate to new audiences, e.g., that will work when language and other 
cultural barriers are involved, actively engage program participants and other 
stakeholders. 

• Credibility of internal evaluation work versus hiring an external evaluator. 

• Evaluating collaborative work — who does it, who gets the credit? 

Personal professional issues: 

• Lack of evaluation knowledge and skills, especially of simple, easy to use, 
evaluation techniques. 

• Lack of time to do evaluations, especially when “crisis” programming occurs. 

• Dealing with the political realities of responding to clients with “clout,” even 
when the request is trivial. 



• Fear about the consequences of identifying lack of success in a program — 
both to Extension administrators and outside funders and decision-makers. 

• Evaluating collaborative efforts — who gets the credit? Is it spmetimes 
politically risky to claim credit when collaborating with external partners? 

• The volume of work done by campus-based faculty that isn’t part of an 
Extension “program” per se, e.g., consulting, advising industry people, etc. 

• Lack of recognition of Extension work, including evaluation, in campus-based 
faculty promotion and tenure decisions. 

• Skepticism about whether evaluations really matter — is the information used 
and useful for anyone other than the staff involved in the program? 

• Is there value/payback for doing evaluations? 

• Preference for having mentors and/or internal or external consultants provide 
technical assistance and aid in designing and conducting program 
evaluations, rather than more formal, “traditional” kinds of general staff 
development on program evaluation. 

• Possibility of creating “affinity” groups — people doing similar programs who 
would share ideas and techniques for evaluating their programs. Could use 
listservs and chat rooms instead of meeting face-to-face. 

Organizational issues; 

• Lack of commitment on the part of Extension administration for genuinely 
supporting program evaluation — in terms of staff development and funding 
forevaluations. 

• Lack of guidelines-unclear expectations of staff and failure by Extension 
administation to articulate the purpose and need for program evaluation, e.g., 
which programs should be evaluated? How will the information gathered be 
used in meaningful ways to secure funding, etc.? 

• Lack of clarity on the purpose(s) of evaluation — how much of it is for program 
improvement, how much for satisfying funders and decision-makers 
(accountability)? How much is for “public relations” and/or program 
marketing purposes? How much is for individual and organizational learning 
about what works and what doesn’t? 

• • Lack of recognition, use of evaluation results in meaningful ways; lack of 

consequences for not evaluating programs. 

• Lack of centralized program planning, program priority and goal-setting. 

• Lack of clarity on what is a “program.” 

• Lack of mechanisms for sharing evaluation results internally, learning from 
each other’s work, sharing ideas. 

• The “disconnect” between campus and field — lack of thorough knowledge of 
what programming is happening — and should be evaluated comprehensively. 

• Lack of assistance in designing and conducting evaluations — inability to hire 
external evaluation expertise due to lack of funding. 
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Summary and recommendations : 



The preceding issues indicate a definite need to re-think the approach to 
program evaluation in the University of Minnesota Extension Service. While a 
number of the focus group participants stated that they perceived program 
evaluation to be an important part of their educator role, there was a general 
concern that organizational commitment, especially in terms of funding, clarity of 
purpose and use of evaluation results, was lacking. As a result, many 
participants thought that at present there are neither rewards or consequences 
related to evaluation, beyond their own satisfaction in learning how they might 
improve their programs or success in gaining additional grant funds for program 
support. The perceptions and attitudes of campus-based faculty were generally 
even less favorable than those of Extension educators. 

Extension administration should address these issues by 

• Clarifying organization commitment to program evaluation and making a 
greater effort to assist in providing technical assistance and funding. 

• Clarifying the purposes and expectations of faculty and staff for doing 
program evaluation and providing guidelines for determining which programs 
should be evaluated. 

• Revitalizing a program planning process that emphasizes incorporating 
evaluation up front. 

• Prioritizing major programs and evaluation efforts. 

• Recognizing and sharing evaluation efforts across the organization. 

• Providing more opportunities for faculty and staff to discuss their work and 
share ideas for improving it, including efficient, effective approaches to 
evaluation that are appropriate to the work. 

• Emphasizing providing more specific technical assistance and other 
resources to faculty and staff when they are designing programs and writing 
grant proposals, than formal staff development. 

• Identifying faculty and staff who are doing exemplary evaluation work and 
rewarding them by providing assistance with their work so that they have 
time to mentor others who are interested in learning from them. 

• Identifying evaluation research and models from other states and 
organizations other than Extension that might be readily adapted for use in 
Minnesota. 
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